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Background / Context: 

The consequences of failing core academic courses during the first year of high school are dire. 
More students fail courses in ninth grade than in any other grade, and a disproportionate number 
of these students subsequently drop out (Herlihy, 2007). As shown in Chicago and elsewhere, 
academic performance in core courses during the first year of high school is the strongest 
predictor of eventual graduation (Allensworth & Easton, 2005). Spearheaded by research from 
Chicago and other large U.S. districts, the use of “early warning” data systems to identify 
students at risk of academic failure and then appropriately intervene is now widely recommended 
(Dynarski et ah, 2009; Heppen & Therriault, 2008; Jerald, 2006) and gaining momentum around 
the country. Identification is a critical first step, but it is only the first step. There is a lack of 
critical information about the types of interventions that can, in fact, get off-track students back 
on track for graduation and improve schools’ graduation rates. 

Algebra failure is of particular concern in high schools across the country. It is considered a key 
gatekeeper for higher-level mathematics course-taking in high school and for college enrollment 
(Adelman, 2006; Gamoran & Hannigan, 2000). Yet, pass rates are consistently low in many 
places. For example, at least 20% of ninth graders in Michigan fail Algebra I (Higgins, 2008). 
Six years after the implementation of an initiative to increase access to algebra, failure rates for 
freshmen in Milwaukee were 47% (Ham & Walker, 1999). In Los Angeles, 44% of ninth graders 
failed Algebra I (Helfand, 2006). In the Chicago Public Schools (CPS), only 13% of students 
who fail both semesters of Algebra I in 9th grade graduate in 4 years, and the largest share of 9th 
grade algebra failures occur in the second semester of the course. Identifying ways that students 
can get back on track is of utmost policy importance. 

Credit recovery is one strategy to address high failure rates. Credit recovery programs offer 
students an opportunity to retake classes that they failed to get them back on track and keep them 
in school (Watson & Gemin, 2008). As schools across the nation struggle to keep students on 
track and re-engage students who are off track, online learning has emerged as a promising and 
popular strategy for credit recovery; over half of respondents from a national survey of 
administrators from 2,500 school districts reported using online learning in their schools for 
credit recovery, with just over a fifth (22%) reporting “wide use” of online learning for this 
purpose (Greaves & Hayes, 2008). 

Despite the growing use of online courses for credit recovery, the evidence base is thin. This 
paper describes the design and initial implementation of a randomized control trial that was 
designed to strengthen the evidence base for online credit recovery. The study tests the (1) the 
impact of online Algebra I for credit recovery against the standard face-to-face (f2f) version of 
the course, and (2) the effects of offering expanded credit recovery options with online algebra, 
relative to business as usual (i.e., summer programming that schools would offer in the absence 
expanded credit recovery efforts). 
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Purpose / Objective / Research Question / Focus of Study: 

As described in the Symposium Justification, this efficacy trial is funded by a grant from the 
Institute of Education Sciences (lES), which provided resources to CPS high schools to 
implement Algebra I credit recovery courses during the summers of 201 1 and 2012 — half online 
and half f2f The study is designed to answer a set of research questions that address the four 
broad aims previously described. This paper focuses on the first aim, which is test the efficacy of 
online Algebra I for credit recovery, compared with standard f2f Algebra I for credit recovery. 

Setting: 

The setting for this study is CPS high schools with the largest number of students who failed 
Algebra I. CPS is the third-largest U.S. district, serving more than 404,151 students in 681 
schools, of which 116 are public high schools and 27 are public charter high schools. The overall 
graduation rate in the district is 65.4%. The study team recruited a total of 15 CPS high schools 
to participate in the study in 2011, and 13 CPS high schools in 2012. These schools were 
selected and recruited for participation because they had the largest number of students who 
failed the second semester of Algebra I in the 2009-2010 school year, did not otherwise have 
existing expanded summer credit recovery programs in place, and were open for summer school 
(e.g., not undergoing renovation). The characteristics of the participating schools are shownin 
Tables 1 and 2. 

Population / Participants / Subjects: 

The target population for this study was first-time freshman who failed Algebra IB. Student 
recruitment focused on freshmen that failed second semester Algebra because they are positioned 
to get back on track in mathematics if they recover the 14 credit early in high school. Students 
were blocked by gender and whether they passed first semester Algebra and then randomly 
assigned to condition on site, on the first day of summer school. 

In year 1 (summer 201 1) we randomly assigned a total of 592 students to 18 pairs of online and 
I2f sections of second semester algebra (Algebra IB) in 15 CPS high schools. Of the 592 
students, 88% were eligible for free/reduced-priced lunch, 9% were eligible for special education 
services, and 38% were female. The students were 57% Hispanic, 36% African American, and 
5% white. Thirty-eight percent of students were known to have failed first-semester algebra. In 
year 2 (summer 2012), we randomly assigned a total of 792 students to 20 pairs of online and f2f 
sections in 13 schools. The cohort 2 student sample was 38% female and 37% were known to 
have failed first-semester algebra. 

Improvement Initiative / Intervention / Program / Practice: 

The study was designed to test whether at-risk students who failed second semester Algebra I 
can achieve higher rates of success if they take an online course for credit recovery than if they 
re-took the failed course in the standard I2f format. The theory of action behind this study is 
represented in Figure 1 in Appendix B. Students fail algebra because they are poorly engaged in 
the class and put in little effort — the strongest predictors of 9th grade course failure are students’ 
attendance and work effort (Allensworth & Easton, 2007). Eow engagement leads students to 
learn little and to subsequently fail. Because they lack an understanding of algebra, they struggle 
in subsequent classes, particularly in mathematics and science. Failure in these classes, combined 
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with failure in algebra, leads students to have insuffieient credits to graduate. As the likelihood 
of obtaining sufficient credits diminishes, students eventually drop out. 

Online credit recovery potentially interrupts this process in two ways. First, the delivery can be a 
more individualized, interactive experience. Furthermore, students receive personal support and 
monitoring from on-site mentors. These characteristics — individualization, interactive pedagogy, 
and personal support — have all been associated with greater engagement and learning 
(Archambault et ah, 2010; Lee & Smith, 1999; Newmann et ah, 1996; Slavin & Madden, 1989). 
Students should be more engaged and more likely to persist in the course, thus more likely to 
learn algebra content and receive course credit. These short-term outcomes should lead to 
improvements in other short-term achievement outcomes, including scores on the mathematics 
exam (that includes an algebra portion) taken in the fall of 10th grade. Better algebra skills 
should also make students more likely to pass their subsequent mathematics and science classes, 
and make greater progress toward graduation. 

The online course used in the study was developed by Aventa Learning, a provider that CPS had 
used in recent years. Students took the course in computer labs at their local high schools, in the 
presence of a trained on-site mentor. They also had an online algebra teacher, provided by 
Aventa. The control condition is the typical f2f Algebra IB course offered in schools 
participating in the study. The course followed the standard CPS Algebra I curriculum and was 
taught by a teacher in each participating school. 

Research Design: 

The study employs an experimental design with on-site student-level random assignment of 
students to either online or f2f Algebra I credit recovery courses in CPS high schools. 
Participating schools were those with Algebra I failure rates sufficient to support the study. The 
focus of student recruitment was on freshmen who failed second semester Algebra because they 
are positioned to get back on track in mathematics if they recover the 14 credit early on in high 
school. Students were blocked by gender and whether they passed or failed first semester 
Algebra and then randomly assigned to condition on site, on the first days of summer school. On- 
site random assignment prevented the inclusion of large numbers of no-shows in the intent-to- 
treat (ITT) analyses. Tables 2 and 3 in Appendix B show the distribution of students by and 
across condition on the blocking characteristics for cohorts 1 and 2. 

Data Collection and Analysis: 

To examine whether participation in credit recovery courses affects the future performance of 
students in mathematics, the study uses achievement and course-taking data as well as annual 
data on students’ dropout/persistence status and, ultimately, graduation status. In this paper we 
focus on the following outcomes for both cohorts: 

1) Measures of short-term learning in algebra, including: 

• A study-administered end-of-course assessment that includes a range of Algebra I 
topics. The assessment was administered to all consenting students participating in 
the credit recovery courses in summers 2011 and 2012. 

• The PLAN assessment algebra sub test scores. 
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2) Grades in the credit recovery courses and credit attainment, collected via 
administrative reeords 

3) Student engagement and attitudes toward math, collected via surveys administered at 
the end of summer session in both years 

4) Enrollment status for the year following participation in the study (2011-12 for cohort 1 
and 2012-13 for cohort 2). 

5) Math and science course performance in grade 10. 

Mathematics achievement outcomes (end-of-eourse assessment of algebra learning and 10th 
grade PLAN algebra subtest) and course performance (grades) are modeled as a fimetion of 
students’ pretest scores and intervention status (online or f2f). Analyses with binary outcomes 
(e.g. passing eredit reeovery course) employ fixed-effects logistic regression models. 

Findings / Results: 

Impact analyses to date show that algebra posttest scores were not signifieantly different by 
eondition in eohort 1, but were higher among students in f2f credit reeovery than online in eohort 
2 (Hedges’ g = -0.23; p =.035). Students in the f2f elasses had higher grades that those in the 
online course in both cohorts (Hedge’s g = -0.40 and -0.58, respectively, both p’s < 0.001). 

While the majority of students in both conditions successfully recovered the algebra eredit, rates 
of credit reeovery were higher in the f2f than online condition; however, the differenee was 
statistically significant only for cohort 2. (Credit reeovery rates for Cohort 1 were 66% online vs. 
70% f2f; p = 0.483, and for Cohort 2 were 68% online vs. 79% f2f; p < 0.001.) Algebra PLAN 
seores were not significantly different by condition in either eohort, nor was the likelihood of 
taking or passing Geometry the following year. (For example, in cohort 1, 86% of f2f students 
and 85% of online students enrolled in second semester Geometry the following spring, and 45% 
of f2f students and 42% of online students passed the eourse; p’s = 0.655 and 0.61 1 for taking 
and passing, respeetively.) Student attitudes were similar by eondition in both eohorts, except 
that the online course was viewed as more difficult in both cohorts, and students’ reported liking 
of and eonfidence in mathematics was higher in the f2f than online condition in cohort 2. 

Conclusions: 

Findings to date suggest that for some short-term outeomes, f2f eredit recovery seems to benefit 
students more than an online eourse. However, most students suecessfully reeovered eredit in in 
both conditions. The course eontent of the online course was more difficult, which may have 
benefitted some students but may have been inaccessible for others. In combination, the findings 
presented in this paper will provide insights about the impaets of online eredit recovery that are 
otherwise unavailable in the researeh literature. 


^ Although randomization does not require covariate adjustments to obtain unbiased estimates of the intervention’s 
effects, the inclusion of covariates strongly related to the outcome, particularly pretest scores, improves statistical 
precision of the parameter estimates (Bloom, Richburg-Hayes, & Black, 2007; Raudenbush, Martinez, & Spybrook, 
2007). 
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